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Abstract. The lower dimensional Busemann-Petty problem asks, 
whether n-dimensional centrally symmetric convex bodies with 
smaller i-dimensional central sections necessarily have smaller vol- 
umes. For i = 1, the affirmative answer is obvious. If i > 3, 
the answer is negative. For i = 2 and i = 3, the problem is still 
open, however, when the body with smaller sections is a body of 
revolution, the answer is affirmative. The paper contains a com- 
plete solution to the problem in the more general situation, when 
the body with smaller sections is invariant under rotations, pre- 
serving mutually orthogonal subspaces of dimensions £ and n — £, 
respectively, so that i + £ < n. The answer essentially depends on 
£. The argument relies on the notion of canonical angles between 
subspaces, spherical Radon transforms, properties of intersection 
bodies, and the generalized cosine transforms. 



1. Introduction 

Let G n> i be the Grassmann manifold of 2-dimensional linear sub- 
spaces of R n , and let volj(-) denote the z-dimensional volume function. 

Question: Suppose that i is fixed, and let A and B be arbitrary 
origin-symmetric (o.s.) convex bodies in W 1 satisfying 

(1.1) vob(AnO <voii(BnO V£eG nii . 

Does it follow that 

(1.2) voln(A) < vol n (5) ? 

This question generalizes the celebrated Busemann-Petty problem, 
corresponding to i — n — 1 [BP] . The latter has a long history, and 

2000 Mathematics Subject Classification. Primary 44AI2; Secondary 52A38. 
Key words and phrases. Radon transforms, intersection bodies, the Busemann- 
Petty problem. 

The research was supported in part by the NSF grant DMS-0556I57 and the 
Louisiana EPSCoR program, sponsored by NSF and the Board of Regents Support 
Fund. 

1 



2 



BORIS RUBIN 



the answer is really striking. It is "Yes" if and only if n < 4; see JG] . 
|K3j . |R3j . |Z2j . and references therein. For 1 < i < n — 2, the problem 
is more intriguing. We call it the lower dimensional Bus emann- Petty 
problem (LDBP). If i = 1, the implication (II. ip — > (II. 2p is obvious 
for all o.s. star bodies without any convexity assumption. In the case 
i = 2, n = 4, an affirmative answer follows from the solution of the 
usual Busemann-Petty problem. For 3 < i < n — 1, a negative answer 
was first given by Bourgain and Zhang [BZj . The proof in [BZJ was 
corrected in [RZJ. An alternative proof was given in [K2j . In the cases 
i = 2 and i = 3 for n > 4, the answer is generally unknown, however, 
if the body with smaller sections is a body of revolution, the answer 
is affirmative; see [GZj . |Zlj . [RZj . It is also known [BZJ, that when 
i = 2 and B is a Euclidean ball, the answer is affirmative provided that 
A is convex and sufficiently close to B. On the other hand [Mi2j, for 
i = 2 and i = 3, there is a small perturbation A of a Euclidean ball, so 
that the implication (jl.ip — > (11.21) is true for arbitrary o.s. star body 
B. Modifications of the Busemann-Petty problem were studied in [K3J, 
[RZJ, [Y], [Zvj ; see also [G] , where one can find further references. 

Main results. In the present paper we give a complete solution to 
the problem stated above, when the body with smaller sections is in- 
variant under orthogonal transformations preserving mutually orthog- 
onal subspaces, say, p and p x , of dimensions £ and n — £ satisfying 
i + £ < n, 1 < i < n. Let us choose the coordinate system in IR n so that 

p = W = © Rej and p L = M. n ~ e = © Re j} where e x , e 2 , . . . , e n 

j=n—£+l j=l 

are the relevant coordinate unit vectors. Without loss of generality, we 
assume £ < n — £, i.e., £ < n/2 (otherwise, the coordinate subspaces 
can be renamed). The case £ = 1 corresponds to bodies of revolution. 
Consider the subgroup of orthogonal transformations 



A star body A is ^-symmetric if 'jA = A for all 7 G K^. Clearly, every 
Kp -symmetric body A is origin-symmetric, that is A = —A . We set 



Every ^-symmetric body in IR n can be obtained, for instance, if we 
take a 2-dimensional body, which is symmetric with respect to coordi- 
nate axes in the plane (ei,e n ), and rotate it about the subspaces W £ 
and R n ~^. A typical example is the (q, £)-ball 



(1.3) 




x = (a/, x") G M n , x' G W 



x" G M! 



(1.4) 



B, 



•n 



{x : \x'\ q + \x"\ q < 1} 



q > 0. 
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The basic idea of our approach is the following. We observe, that the 
relative position of a subspace £ G G n ^ with respect to the coordinate 
subspace M. 1 is detemined by m = min(z, £) canonical angles o>i, . . . , oj m \ 
see, e.g., [Cj. We define 

(1.5) G t nii ={£eG nii :u 1 = ... = u m } 

to be the submanifold of all £ G G n ^ such that all canonical angles 
between £ and 1R £ are equal. The structure of the set G e ni can be 
understood as follows. Let Ai = cos 2 u;i, . . . , A m = cos 2 a; m . These are 
eigenvalues of the positive semi-definite matrix 

r'P R er if i < 
a'P^a if i > 

where r and o denote arbitrarily fixed orthonormal frames which span 
£ and M. e , respectively; r', a', P R «, and stand for the corresponding 
transposed matrices and orthogonal projections. We arrange Ai, . . . , A m 
in non- increasing order and regard A = (Ai, . . . , A m ) as a point of the 
simplex 

(1.7) A m = {A : 1 > Ai > A 2 > • • • > \ m > 0}. 

The edge {Ai = . . . = A m } of this simplex corresponds to G l n i . 
Our main results are the following. 

Theorem 1.1. Let 1 < £ < n/2, i+£ < n, and let Abe a Kg- symmetric 
star body in W 1 . 

(a) If 1 < i < £, then the implication 

(1.8) vok(A n < vok(B HO V£ G G £ ni =^ vo\ n (A) < voln(S) 
is true for every o.s. star body B. 

(b) If % — I + 1 or i = £ + 2, then U.8\) holds for every o.s. star body 
B provided that A is convex. 

Theorem 1.2. If i > £ + 2, and B = B n i t = {x : \x'\ 4 + |x"| 4 < 1}, 
then there is an infinitely smooth Ki-symmetric convex body A, such 
that voh(A H < voli(B n for all £ G G n>i , but vol n (A) > vo\ n (B). 

Some comments are in order. 

1°. It might be surprising, that to make a positive conclusion in 
Theorem II. 1[ we do not need all i-dimensional central sections, as sug- 
gested in the original problem. It suffices to consider only sections 
having equal canonical angles with respect to ~R e . More advantages of 
our method of canonical angles are described in Remark 12.61 
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2°. The condition i + £ < n in Theorem 11.11 excludes the situation 
when dim(£ n R e ) > 1 for all £ G G^; see Remark EH We actually 
assume 

in (a): i < min(£, n — £); 

in (b): £ < (n - l)/2, if z = £ + 1, and £ < (n - 2)/2, if z = £ + 2. 
Regarding (a), the situation, when inequalities i + £ > n and i < £ 
hold simultaneously, is impossible, because in this case £ > n/2, that 
contradicts our initial convention. Regarding (b), a simple examination 
shows that the following cases, which are admissible when i + £ > n, 
are not presented in Theorem 11.11 

(i) n = 2£, when i = £ + 1; 

(ii) n = 2£ and n = 2£+l, when i = £ + 2; 

The validity of the implication (11.11) — > (11. 2p in (i) and (ii) is an open 
problem. After several attempts to attack it, we have got an impression 
that the difficulties here have the same nature as those in the original 
LDBP for z = 2 and 3. 

3°. Another intriguing open problem is to check the following 
Conjecture. In the case (b) of Theorem \l.l\ i.e., when i = £+ 1 or 
i = £ + 2, there exist a non-convex K e - symmetric body A and an o.s. 
star body B so that 

vol, (A n < vol, (B H for all £ G G n>i (not only for £ G G^J, 

but vol n (A) > vol n (B); cf. [G, Theorem 8.2.4] for n = 3, £ = 1. 

The paper is organized as follows. In Section 2 we obtain new lower 
dimensional representations for the spherical Radon transform of Kf 
invariant functions; see Theorem 12.21 and Corollary 12.31 These results 
are used in Section 3 to prove Theorem 11.11 Theorem 11.21 is proved in 
Section 4, where we invoke some facts on intersection bodies and the 
generalized cosine transforms. The concept of intersection body was 
introduced by Lutwak [Luj and extended by Zhang |Zlj and Koldobsky 
[K2] to lower dimensional sections. Useful information about these 
objects can be found in |K3j . |Milj . [R4j . 

Acknowledgements. The author is grateful to Professors Alexander 
Koldobsky, Erwin Lutwak, Deane Yang, and Gaoyong Zhang for useful 
discussions. 

Notation: We use the standard notation 0(n) and SO(n) for the 
orthogonal group and the special orthogonal group of lR n endowed with 
the invariant probability measure. For 1 < i < n, we denote by G n ^ 
the Grassmann manifold of i-dimensional subspaces £ of lR n ; d£ stands 
for the 0(n)-invariant probability measure on G n /, S 1 ™ -1 is the unit 
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sphere in R n ; o n _\ = 2n n ^ 2 /Y{n/2) is the area of S n ~ l ; e 1; e 2 , . . . , e n 
denote the coordinate unit vectors; 9Jl nji is the space of real matrices 
having n rows and i columns. For X G ^fl n ,ii X' denotes the transpose 
of X, Ii is the identity i x i matrix; 

Km = {r G Wl n4 :T , r = I i } = 0{n)/0{n - i) 

is the Stiefel manifold of orthonormal i-frames in R n . For r G V n> i, {r} 
denotes the i-dimensional subspace spanned by r. All vectors in R n 
are interpreted as column-vectors. 



2. The Spherical Radon Transform of ^-Invariant 

Functions 

For functions f{9) on S*™ -1 and on G n ^ we define the spherical 
Radon transform (/?;/)(£) and its dual (R*ip)(9) by 



(2.1) (Rif)(0= J f(0)d<9, 



(R*<p)(9) = J <p(QdoZ, 
&6 



where measures d^9 and do^ are normalized so that Ril = and 
R*l = 1. The corresponding duality relation has the form 



(2.2) 



{RiMMO^ 



f{9){R^){9)d9 



and is applicable whenever either side is finite for / and tp replaced by 
| f\ and \(p\, respectively; see [He], |R2j . 

In this section we obtain explicit "lower dimensional" expressions for 
Rif when / is ^-invariant. We remind that 



an— £ 



n-i 



-J' 



j=n-£+l 



(2.3) R n 

1 < I < n — 1, and set 

(2.4) a = [e n _ m 
Every 9 G S 1 ™ -1 is represented in bi-spherical coordinates as 







(2.5) 



m sinw 

U COS Co" 



ueS 



n-t-1 



71 



v G S'- 1 , 0<u<- 



so that dd = sin 



n-£-l 



cos^ui dudvdu; see, e.g., |VK] . Clearly, cos' 
9'aa'9 = 9'P R t9, where P R e denotes the orthogonal projection onto 
The following statement is an immediate consequence of (12.51) . 
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Lemma 2.1. A function f on S"^ 1 is K^-invariant if and only if there 
is a function f on [0, 1] such that f(9) = fo(t), where t x l 2 = (O'V^tO) 1 / 2 
is the cosine of the angle between the unit vector 9 and the coordinate 
subspace M. e . Moreover, 

tt/2 

J f(6)d6 = c J sin^-^cos^/otcos 2 ^)^ 



S n-1 



(2.6) 



£ J t £/2-l (1 _ t) (n-£)/2-l /c 



(t) dt, 



C — (T£_i(T n _£_i. 



Theorem 2.2. Let 1 < i,l < n — 1; m — min(i, £). Let . . . , oo m 

be canonical angles between the subspace £ G G nii and the coordinate 
plane M. e , 

(2.7) A = diag(Ai, . . . , A m ), Ai = cos 2 ^, . . . , A m = cos 2 cu m . 

Suppose that f is a Ke-invariant function on S" 1 ' 1 , so that f{6) = 
fo(t), t = cos 2 u, where uj is the angle between 6 and M*. Then the 
Radon transform Rif has the form = F(\), where 

v'Xv 



(2.8) F(X): 



cri-e-i 



dv 



(v'Xvy/ 2 - 1 



t e ~' 2 -\v'\v-t) 



(i-£)/2-l 



f (t)dt 



if i > I, and 

(2.9) F{\) = J f {v'\v) dv 

S i-i 

if %<i. 
Proof. We set 

Pi z 



Ii 




e V n ,t, {pi} = © Re,, 
j'=i 







g 



and let G SO(n) be a rotation that takes the subspace {p^ to 
£ G G^j. Then (set = p^rj) 

(RJ)(0= J Waa'9)d^= f foirj'p'^a'p^drj, 

Sn-l n £ S i-1 

S^ 1 being the unit sphere in {pi}. Let 



(2.10) u = p ( o-- 



Ui 

u 2 



eK/, u 1 =p' j u=p' i p'^e'>m i/ , u 2 em n _ i/ . 
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Then rj'u = rfui, and we have 

(2.11) (i2i/)(0= / foWuu'r ] )dr ] = J /ofaW^) dtj. 

gi-i gi-i 

Consider the case £ < i and write u\ in the form (cf. [Mu, p. 589]) 



7 G SO(i), p e 



h 




where r is a positive semi-definite £ x £ matrix defined by 
(2.12) r = u[ui = u'pip[u = <j' p^pip'ip'^a = a'P^a. 

Hence, 



W)(0 



fa (rflfPe t p'i i'v) dr) — / faiC'Perp'tO d( 



Since £ < i, then {p e } C and we can write £ in bi-spherical 

coordinates 



c 



D COS'?/' 

w sin ip 



veS 



e-i 



weS 



i-e-i 



< ij) < 



71 



so that d( = cos e 1 ip sin* 1 ^ dvdwdtp. This gives p'^C = v cos?/', and 
therefore, 

tt/2 

W){0 = Oi-t-i J dv J fo(v'rvcos 2 ip)cos e - 1 i) sin^' 1 ip dtp 

Sfi-i o 



(2.13) 



dv 



si- 



f _^/ t '/- 1 („V B -t)(«)/«/oW«. 



Finally, we diagonalize r = a'P^cr by setting r = "/Xj, where 7 G 0(£) 
and A = diag(Ai, . . . , A^). Changing variables, we obtain (12. 8p . 

Consider the case £ > i. We replace u\ in (12.111) by p'iP^cr from 
(I2.10p and let r G Km an arbitrary orthonormal i-frame in £. We 
can always choose p^ so that pffi = r. Then U\v! x = p' i p'^T& p^Pi = 
t'oo't. The i x i matrix s = t'oo't is positive semi-definite and can 
be diagonalized as above. Hence, (12. lip yields 



WHO 



fo(r]' sr]) dr] = / f (r}'\rj)dr} 



as desired. 



□ 
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Corollary 2.3. If all canonical angles in Theorem \2.2\ are equal, that 
is, Ai = ■ ■ ■ = A m = A ; then = F(X), where 

A 

(2.14) F(A) = a ^- 1 jt^-\X-tf-^f (t)dt 

o 

if i > £, and 

(2.15) F(A) = o- i _ 1 / (A) 
i/ i < £. 

Remark 2.4. If i + £ > n, then every £ G G n> i has at least one- 
dimensional intersection with M. e . It means that some canonical an- 
gles between £ and M £ are necessarily zero and therefore, some of the 
eigenvalues Ai, . . . A m equal 1. It follows that for i + £ > n, equalities 
( 12. 14)) and (I2.15P are available only for A = 1. This situation is not 
favorable for our purposes, because we will need (12. 14j) and (12 . 1 5|) to be 
available for all A G (0, 1). The latter is guaranteed if i < n — £, when 
we have "sufficiently many" z-dimensional subspaces with the property 
dim(£ n R e ) = 0. 

Corollary 12.31 motivates the following 

Definition 2.5. We denote by G l n i the submanifold of alii -dimensional 
subspaces £ with the property that all canonical angles between £ and 
Mr are equal. 

Remark 2.6. It is known, that the Radon transform is overdetermined 
if the dimension of the target space is greater than the dimension of 
the source space. If / is .fQ-invariant and i < n — £, then, by Corol- 
lary 12.31 and Remark 12.41 the overdeterminicity can be eliminated if 
we restrict (i?j/)(£) to £ G G l ni . Here one should mention the general 
method of the kappa-operator, which allows us to reduce overdeter- 
minicity by invoking the relevant permissible complexes of subspaces; 
see, e.g., |CCR] and references therein. The advantage of our method 
of canonical angles, which is applicable to the particular case of Ki 
-invariant functions, is the following. If i > £, then to recover / from 
Rif, it suffices to invert a simple Abel integral (12.141) . If i < £, then / 
expresses through Rif without any integro-differential operations. 

3. Ki -Symmetric Bodies and Comparison of Volumes 

3.1. Preliminaries. An origin-symmetric (o.s.) star body B in M. n , 
n > 2, is a compact set with non-empty interior such that tB C B\/t G 
[0, 1], B — —B, and the radial function Pb(0) = sup{A > : X9 G B} 
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is continuous on S 1 " -1 . The Minkowski functional of B is defined by 
= min{a > : x G aB}, so that \\0\\b = Pb 1 ^)- An o.s. star 
body B is called infinitely smooth if p B (9) G C™ en (S n ~ l ). 
If £ G Cr n> j, 1 < i < n, then 

(3.1) voi 4 (snO = r 1 / ^(0)d c e = r 1 (i2 i/O i)(O. 

Similarly, vol n (S) = n" 1 Jcn-i Pb(^) 

Problem. Lei i be a fixed integer, 1 < i < n — 1. VFe wonder, for 
which o.s. star bodies A and B in W 1 the inequality 

(3.2) voL(AnO < voL(5n£) V£GL7 nji 

(3.3) vol* (A) < vol n (5). 

For i = 1 an affirmative answer is obvious. Unlike the question in 
Introduction, here we do not assume a priori that A and B are convex. 
The reason is that (13.21) — > (13.31) may be valid without any convexity 
assumption (see Theorem ll.il (a)) and we want to understand how the 
convexity comes into play. 

Below we study this problem when the body A with smaller sections 
is symmetric with respect to some mutually orthogonal subspaces, say, 
p and p x , of dimensions £ and n — £, respectively. We fix the coordinate 

system so that p = R e = © R ej and p x = R n ~ e = © Re,-. Then 

j=n-£+l ' j=l 

Kf A = A, where Kg is the group (11.31) . An o.s. star body with this 
property is said to be Ki -symmetric. 

By Lemma T2.1[ the radial function Pa{&) of a ^-symmetric body A 
is completely determined by the angle u between the unit vector 9 and 
the subspace WL e . Hence, we can set 

(3.4) p A {6) = p A {t), t = cos 2 u; = 0'P R <0. 

By Theorem 12 .2\ the Radon transform (_Rj/)(£), £ G G Uj i, of every 
Ki -invariant function / is actually a function of the canonical angles 
between £ G G Hji and 1R £ . Restricting (i?j/)(£) to £ G G l ni (see Defi- 
nition [23]), we can remove overdeterminicity of R%f. As we shall see 
below, the lower dimensional Busemann-Petty problem inherits this 
overdeterminicity, and the latter can be eliminated in the same way by 
considering sections by subspaces ( 6 G' nj only. 
We will need the following auxiliary lemmas. 
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Lemma 3.1. The group Kg preserves canonical angles between £ G G n> i 
and R e . 

Proof. The proof relies on (11.61) . Let first £ < i, £ = {r}, r G V n ^. 
It suffices to check that for every 7 G fQ , matrices r = o'tt'o and 

a 



a'^rr'y 'a have the same eigenvalues. Let 7 



/? 



where 



a GO(n-f), /3 G 0(£). Multiplying matrices, we have 7V = a/?'. Hence, 
r 7 = f3a'rr'a[3' = (3r(3' . Since /3r/3' and r have the same eigenvalues, 
we are done. 

If I > i, we compare eigenvalues of matrices s = r'aa'r and s 7 = 
r'^'aa'^r. These matrices coincide, because, as we have already seen, 
7V = cr/3', and therefore, s 7 = r'ofi' fio'r = r'aa'r = s. □ 

Definition 3.2. (iQ-symmetrization) Given an o.s. star body B in 
W 1 , we define the associated Kg-symmetric body Bq by 

i/i 



(3.5) 



p l B {lO)d 1 



Lemma 3.3. vol n (-B ) < vo\„(B). 

Proof. By the generalized Minkowski inequality, 



vo# B (Bo) 



p i B { 1 9)d 1 



n/i \i/n 



d6 



< 



gn-1 
1 

n 



i/n 



p B ( 7 9)d6 rf 7 = volf (B), 



and the result follows. 

Lemma 3.4. Let A and B be o.s. star bodies in M. n , 1 < 
A is Kg-symmetric, and 

(3.6) voL.(An£) <voL.(5nO V£ G G e n>i , 

then vo\i(A n < voli{B n /or a// £ G G^. 



< n 



□ 
1. // 



Proof. Fix £ G G„ j. By Lemma [37TI 7^ G (7„ f for every 7 G -fQ. Owing 
to (J3D, vol l (An 7 < voL( J Bn 7 or (^KtO < (RiPh)(lO for all 
7 G Kg. Integrating this inequality in 7 and taking into account that 
Ri commutes with orthogonal transformations, we obtain 

(3.7) 



□ 



{RiP\m < Ri / P l B h0) d 1 (0 = (RiP l Bo )(0- 



K e 



This implies vok(A n f) < voli(5 n f )• 
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3.2. The case i < min(£, n — £). The following proposition represents 
part (a) of Theorem 11.11 

Proposition 3.5. Let 1 < i,£ < n — 1; i < min(£, n — £). If A is a 

Kit-symmetric body in W 1 , then the implication 

(3.8) vo\i{A n < vol i ( J B D Vf € G e n>i => vo\ n (A) < vo\ n (B) 
is true for every o.s. star body B. 

Proof. By Lemma 13.41 for all £ G G n i we have 

vol,(A n e) < vol^So n £) or (i^XO < (^O(e). 

Hence, by ( 127L5D and (E3D, P^(A) < p J B() (A) for all A G (0,1) (see 
Remark £1, and therefore, p A (0) < p Bo (6) for all G S 11 ' 1 . By 
Lemma [3.31 it follows that vol n (A) < vol n (B ) < vol n (B). □ 

3.3. The case £ < i < n — £. We will need some sort of duality which 
is a one-dimensional analog of (12.21) and serves as a substitute for the 
Lutwak's connection |Luj between the Busemann- Petty problem and in- 
tersection bodies. According to (12.141) . the Radon transform (R i p b A )(£), 
restricted to £ G G e ni , is represented by the Abel type integral 

A 

(3.9) (J^)(£) = _*_ jt^-\X-tf-^- l p\(t)dt, 



Cl = (7f_l/2, 

where A 1 / 2 G (0, 1) is the cosine of the canonical angles between £ and 
M. e (we remind that these angles are equal when £ G G n ^ and (13. 9p is 
available for all A G (0, 1); see Remark |2.4p . Denote the right-hand side 
of (13. 9p by (I + p l A )(\) and define the dual integral operator 

l 

(s.io) (i^)(t) = ci ^ J(\- tf-^m^ 

t 

so that 

i i 

(3.H) f (J+&XA) ^(A) d\ = J p\(t) (J_V»)(0 dt. 

o o 
Expression ( 13.101) resembles the classical Riemann-Liouville integral 

i 

i r , a _! 



(3.12) (J^)(t) = — / <?(A)(A - t)"" 1 dA, a > 0. 
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Lemma 3.6. Let l<£<i<n — £ and suppose that A is a Kg 
-symmetric body in R n . If there is a non-negative function g on (0, 1), 
which is integrable on every interval (S, 1), < 5 < 1, and such that 

(3.13) (l-tjM^rWH^W, a = 1 ^T' te (°>v> 

then the implication 

(3.14) voU(A n < vo\ t (B n V£ G G e n>i vol n (A) < vol n (5) 
/ioWs /or even/ o.s. star frocfo/ 5. 

Proo/. By fl23|) . 

l 

voi n (A) = i y p5(0) ^ = C2 y pko^-^i - f)^/ 2 - 1 dt, 

S"- 1 

c 2 = a l - 1 a n - t - 1 /2n. 
Hence, owing to fl3~TUD . (ETIIj) . and (15331) . 

i i 

voi n (A) = C2 y A( t) ^(^)(*)*=| y f? A {W-m)dt 

o o 
i 

(3.15) = — f (I+^ A )(X)ip(X)dX, m = Xt/2 ~X X) >0- 

ci J r(a) 
o 

If voL,(A n < voli(5 n G G^, then, by Q and (|3"77]L 

(I + &)(A) = (ifcpkXO < (Vflb)(0 = (hPB )W, 
and therefore, 



vol n (A) < ^ /(/ + p i Bo )(A)^(A)rfA 



Cl 



^2 




1 





1 

n 

gn-1 



Now Holder's inequality yields vol n (A) < vol n (S ), and the result fol- 
lows by Lemma [3.31 □ 



THE LOWER DIMENSIONAL BUSEMANN-PETTY PROBLEM 



13 



Up to now, the ^-symmetric body A with smaller sections was 
arbitrary. To handle the case i > £, we additionally assume that A 
is convex. The following lemma enables us to reduce consideration to 
smooth bodies. 

Lemma 3.7. Let A and B be o.s. star bodies in R n . If the implication 

(3.16) vol, (A n < vol, (B n £) V£ G G l n i => vo\ n (A) < voln(fl) 

is true for every infinitely smooth K^-symmetric convex body A, then 
it is true when A is an arbitrary K^-symmetric convex body. 

Proof. Given a ^-symmetric convex body A, let A* = {x : \x ■ y\ < 
1 Vy G A} be the polar body of A with the support function Ha* {%) = 
msix{x-y : y G A*}. Since Ha*{-) coincides with Minkowski's functional 
|| ■ |U) then Ha*{-) is ^-invariant, and therefore, A* is ^-symmetric. 
It is known |Schnt pp. 158-161], that any o.s. convex body in R™ can 
be approximated by infinitely smooth convex bodies with positive cur- 
vature and the approximating operator commutes with rigid motions. 
Hence, there is a sequence {A*} of infinitely smooth ^-symmetric con- 
vex bodies with positive curvature such that Ha* (0) converges to Ha* iff) 
uniformly on S n ~ l . The latter means, that for the relevant sequence of 
infinitely smooth ^-symmetric convex bodies Aj = (A*-)*, 

lim max | \\9\\a ■ — II^IUI = 0. 

This implies convergence in the radial metric, i.e., 

(3.17) lim max \p A M - p A {9)\ = 0. 

Let us show that the sequence {Aj} in (13.1 7p can be modified so that 
Aj C A. The idea of this argument was borrowed from [RZj . Without 
loss of generality, assume that pa(Q) > 1- Choose Aj so that 

\paM ~ pa(o)\ < -^-7 wgs^ 1 

3 + 1 

and set A', = -A^Aj. Then, obviously, Pa'.(@) — > Pa{@) uniformly on 
S 71 " 1 as j — > oo, and 

j j \ 

PA> = ——{PA, < —t(PA + ——) < PA- 

j j+1 J+l 3 + 1 

Hence, Aj C A. 

Now suppose that vol, (A n £) < vol^-B n() V( 6 G e n i . Then 
this is true when A is replaced by A'j, and, by the assumption of the 
lemma, vol n (A'j) < vo\ n (B). Passing to the limit as j — > oo, we obtain 
vol n (A) < vol n {B). □ 
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The next proposition gives part (b) of Theorem 11.11 

Proposition 3.8. Let A be a Kg- symmetric convex body in W l , and 
let 2<i<n-£. If 

i = £+l (in this case £ < (n - l)/2) 

or 

i = £ + 2 (in this case £ < (n - 2) /2), 
then the implication 

(3.18) vok(A n < voli(B n()V(G G e n>i voln(A) < vol n (B) 
ZioZds /or ever?/ o.s. star froei?/ 5. 

Proof. By Lemma 13.71 we can assume G C 00 ^™ -1 ). If i = £ + 2, 
then (13TT31) becomes (I - 1)^' 2 p n A ~\t) = £ g(\) d\, which implies 

9(t) = -j t [(l-t)^p^(t)]eL\0,l). 

To check that g is nonnegative, we set t = 1 — s, r(s) = s 1 / 2 p J 4(l — s), 
s = sin 2 u;, and get 

o(l - s) = -^-[r n_l '(s)] = (n - 2)r"- i - 1 (s)r / (s). 

If = u sinw + wcoscj G S" 1-1 , u G S"^" 1 c M 71 "', v G S^ 1 C M £ , 
and P UjV is a 2-plane spanned by u and i>, then A D P U)t) is a convex 
domain, which is symmetric with respect to the u and t> axes. Since 
s = sin 2 co>, then r(s) = s 1 ^ 2 pa(1 — s) is non-decreasing, and therefore, 
r'(s) > 0. This gives g(l — s) > 0, s G (0, 1), or, equivalently, g(t) > 
for all < t < 1. Now the result follows by Lemma [3. 61 

Let i = £ + 1. We set x A (t) = (1 - t)^' 1 ^ 2 ^ (t) and reconstruct 
g(t) from (I3.13P using fractional differentiation as follows: 

i 

1 d f 

g{t) = ( s -t)- 1/2 x A (s)ds (setp= 1-t, q= 1 - s) 

y/Tc at J 

t 
p 

= -LA [{p-q)- 1 / 2 M A (l-q)dq 

V 71 " d P J 
o 

i 

J {I - 7])~ 1/2 x A {l - pi]) drj . 



1 d 

7T dp 



pi/a 



o 
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This gives 

1 



ir dp J ^ v (l - rj) 

The last integral is a non-decreasing function of p, and therefore, the 
derivative of it is non-negative. Hence, g(t) > for all < t < 1 and, 
by Lemma [3.61 we are done. □ 

4. The negative result 

The proof of the negative result in Theorem 1 1 . 2 1 relies on Koldobsky's 
generalizations of the Lutwak's concept of intersection body (see [K2J, 
[Luj ) and properties of the generalized cosine transforms [R4J. 

We remind basic facts. The generalized cosine transform of a func- 
tion / on S 1 ™ -1 is defined by 

(4.1) (M a f)(u)= ln (a) [ f{6)\6-u\ a ~ l d9, 
. , , , a n _ir((l-a)/2) 

(4.2) 7n ( a ) = _ I _ ]7 ___ ; Rea>0, a^l, 3, 5, . . . . 

The integral (14.11) is absolutely convergent for any / G L 1 (S' n_1 ). If 
/ is infinitely differentiate, then M a f extends as meromorphic func- 
tion of a with the poles a = 1,3,5,.... The following statement is a 
consequence of the relevant spherical harmonic decomposition. 

Lemma 4.1. [Rl] Let a, f3 G C; a, (3 ^ 1, 3, 5, . . . . Ifa + f3 = 2-n 

andfeC™ en (S n - 1 ), then 

(4.3) M a M p f = f. 

If a, 2 — n — a ^ 1,3,5,..., then M a is an automorphism of the space 
C^ , en (5' n ~ 1 ) endowed with the standard topology. 

We will also need the following statement, which is a particular case 
of Lemma 3.5 from |R4|. 



Lemma 4.2. Let f G C^^S 71 ' 1 ). Then 

(4.4) (RiM^fm = c(R n ^m x ), £ G G n , 

where c = c(n, i) is a positive constant. 
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Definition 4.3. |K2] An o.s. star body K in W 1 is a k -inter section 
body if there is a non-negative finite Borel measure \i on S n ~ x , so that 
for every Schwartz function <f>, 



oo 

k-1 



\\x\\K<f>(x) dx = / / t k - l <f)(te)dt dfi(6), 
Js"- 1 '-Jo J 

where <j) is the Fourier transform of 4>. We denote by 1% the class of 
all k -intersection bodies in R™. 

The following equivalent definition is a particular case of the more 
general Definition 5.4 from |R4] . 



Definition 4.4. An o.s. star body K in M" is a k -intersection body if 
there is a non-negative finite Borel measure p on S^ 1 , so that p\ = 
M^V, i-e., (p k K ,<f) = (frM 1 -"?) for any <p G C^S^ 1 ). 

The next proposition plays a key role in the proof of the negative 
result in this section. 

Lemma 4.5. Let B be an infinitely smooth Ki -symmetric convex body 
with positive curvature. If B ^ Z^_ i; then there is an infinitely smooth 
Ki-symmetric convex body A such that voL,(v4n£) < volj(.Bn£) V£ G 
G U:i , but vol n (A) > voln(-B). 

Proof. By Definition 14.41 with k = n — i, there is a function ip in 
C^ en (S n ~ l ) , which is negative on some open set Q C S 1 ™ -1 and such 
that p n B % = M 1+l ~ n p. Since B is Ki -symmetric and operators M a 
commute with orthogonal transformations, then ip is Kg -invariant and 
p < on the whole orbit Qi = KiQ. Choose a function h G C^, en (S n ~ r ) 
so that h^O, h{6) > if 9 G fi* and h(9) = otherwise. Without loss 
of generality, we can assume h to be Ki -invariant (otherwise, we can 
replace it by h{9) = f K h( r yO)d r y). Define an origin-symmetric smooth 
body A by p l A = p l B — eM 1 ~ t h, e > 0. Clearly, A is Ki -symmetric. If 
e is small enough, then A is convex. This conclusion is a consequence 
of Oliker's formula [Olj . according to which the Gaussian curvature of 
an o.s. star body expresses through the first and second derivatives of 
the radial function. Since by g3D, ( J R i M 1 " i /i)(0 = c{R n .ih)(^) > 0, 
then Rip\ < Rip B . This gives vol, (A n < vol, {B n f) V£ G G n>i . 
On the other hand, by (14.31) . 

(Pb~\ Pb - P\) = e(M 1+l - n p, M^h) = e(p, h) < 0, 

or {p B ~~\ p l B ) < (p B ~\ p\)- By Holder's inequality, this implies vol n (5) < 
vol„(.l). " " ' □ 
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Consider the (g, £)-ball B™ e = {x = (x',x") : \x'\ q +\x"\ q < 1}, where 

TL — £ TL 

x' e R n ~ e = © R ej , x" el'= © Rej. 

j=l j'=n— £+1 

Lemma 4.6. (cf. [K3, Theorem 4.21]) Let 

(4.5) = {(x 1 , y) : \x'\ q + \y\ q <1, x' eR m ,y e R}. 

Ifq>2andm>k + 3, then £ Z™ +1 . 

Lemma 4.7. If q > 2 and £ + 2 < i <n — 1, then B™ e <£ l^-i- 

Proof. Suppose the contrary and consider the section of B™ e by the 
(n — £+1) -dimensional plane rj = Re n ©IR n_£ . By Proposition 3.17 from 
|Mi2j (see also more general Theorem 5.12 in |R4j ) B™ e nr] e in 
rj, but this contradicts Lemma T4.6[ according to which (set m — n — t) 
Byg fl rj is not an (n — ^-intersection body when i > i + 2. □ 

For q = 4, Lemmas I4.5I and I4.7I imply the following negative result. 

Proposition 4.8. If£ + 2<i<n — 1, £/ien i/iere zs an infinitely 
smooth K \- symmetric convex body A such that 

voh(AnO < vo^s^nO veeG nil , vol n (A) > vol n (£^). 

This is just Theorem 11.21 
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